Image enhancement methods

ABSTRACT

Methods of image enhancement are disclosed. In one aspect, the method of image enhancement is for use with an image capture device, such as a security document reader, for the attenuation, separation or reduction of reflections from objects, such as security documents.

Security documents such as passports, identification cards, nationalhealthcare cards, driver's licenses, entry passes, ownershipcertificates, financial instruments, and the like, are often assigned toa particular person by personalization data. Personalization data, oftenpresent as printed images, can include photographs, signatures,fingerprints, personal alphanumeric information, and barcodes, andallows human or electronic verification that the person presenting thedocument for inspection is the person to whom the document is assigned.There is widespread concern that forgery techniques can be used to alterthe personalization data on such a document, thus allowingnon-authorized people to pass the inspection step and use the documentin a fraudulent manner.

A number of security features have been developed to help authenticatethe document of value, thus assisting in preventing counterfeiters fromaltering, duplicating or simulating a document of value. Some of thesesecurity features may include overt security features or covert securityfeatures. Overt security features are features that are easily viewableto the unaided eye, such features may include holograms and otherdiffractive optically variable images, embossed images, andcolor-shifting films. In contrast, covert security features includeimages only visible under certain conditions, such as inspection underlight of a certain wavelength, polarized light, or retroreflected light.One example of a laminate that includes both overt and covert securityfeatures is 3M™ Confirm™ Laminate with Floating Image Technology, whichis commercially available from 3M Company based in St. Paul, Minn. Thissecurity laminate may be used with security documents, such asidentification cards, badges and driver licenses, and assists inproviding identification, authentication and to help protect againstcounterfeiting, alteration, duplication, and simulation. Another exampleof a laminate that includes both overt and covert security features isillustrated in U.S. Pat. Publication No. 2003/0170425 A1 “SecurityLaminate,” (Mann et al.).

In recent years there has been widespread adoption of automated readingof security documents at border entry points and other situations wherethe identity of a document holder requires verification. Automatedreading ranges from an optical scan of OCR-readable data to theinterrogation of an RFID chip within a passport or identification card,which may then involve further checking by an operator or verificationby an automated system such as an e-passport gate as found in majorairports. Data may also be contained in a magnetic strip or transferredwirelessly depending on the format of the document in which identityinformation is contained.

Optical reading of a security document is typically carried out withdocument readers using one or a combination of visible, infrared andultraviolet light, depending on the information being retrieved. Oftenovert and covert optical security features, such as those discussedabove, are included within security documents to allow the documentitself to be authenticated as genuine. As discussed, covert securityfeatures may only be visible under certain illumination, such asinfrared or ultraviolet light, or may, such as with a hologram, providevariable information when illuminated from different directions. In eachcase the security document is typically read by placing the document ona glass platen of a document reader, such that the information containedon the portion of the document in contact with the platen is illuminatedfrom within the document reader. Light reflected by the document isreflected back into the reader and processed to form an image of theinformation (e.g. text or covert or overt security features) required.The quality of the image captured is affected greatly by the manner inwhich the document reflects the incident light.

A variety of security readers are known in the art. For example, U.S.Pat. No. 6,288,842, “Security Reader for Automatic Detection ofTampering and Alteration, (Mann) discloses a security reader for readingand processing information about security laminates. One example of apassport reader is commercially available from 3M Company based in St.Paul, Minn., as the 3M™ Full Page Reader.

Image enhancement by removal of unwanted reflections in image capturedevices is disclosed in U.S. Pat. No. 7,136,537, “Specular Reflection inCaptured Images,” (Pilu et al.). In order to remove specularreflections, two images are taken, one containing specular reflectionsand one where such reflections are absent. These images are blendedtogether to create an image with reduced specular reflection, allowingunderlying features to be seen. The apparatus used to achieve thiseffect is provided with an adjustor that is able to vary the amount ofspecular reflection appearing in the final image. Images are taken withone or more strobes or flashes from various directions relative to theobject being imaged, and relies on each image having an absence of glarepatches seen in another image. Such a method therefore takes intoaccount reflections generated by ambient light conditions, and is notsuitable for use in a document reader, for example, where illuminationis well controlled and reflection features are generated by artefacts inthe document being imaged, rather than artefacts generated by variationsin ambient illumination.

SUMMARY

One aspect of the present invention provides an image enhancement methodfor an image capture device. This method comprises: illuminating anobject placed on, in or adjacent to the image capture device andcapturing an image of the object from a first position to obtain a firstset of raw pixel data; illuminating the object placed on, in or adjacentto the image capture device and capturing an image of the object from asecond position, to obtain a second set of raw pixel data, wherein eachpixel in the second set of raw pixel data corresponds to a pixel in thefirst set of raw pixel data representing a point on the object;calibrating each of the first and second sets of raw pixel data using aset of image calibration pixel data to create a first set of image pixeldata and a second set of pixel image data; and calculating a first setof final image data by: comparing the first and second sets of imagepixel data; for pixels representing the same point on the object,selecting the pixel with the lowest pixel intensity; and including saidpixel in the first set of final image data.

The above summary of the present invention is not intended to describeeach disclosed embodiment or every implementation of the presentinvention. The Figures and the detail description, which follow, moreparticularly exemplify illustrative embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be further explained with reference to theappended Figures, wherein like structure is referred to by like numeralsthroughout the several views, and wherein:

FIG. 1 is a schematic side view of a document reader in which anembodiment of the method of the present invention is carried out;

FIG. 2 is a schematic side view of one type of optical defect in asecurity document giving rise to a specular reflection;

FIG. 3A is a schematic illustration of an image of a passport bio-datapage illuminated from a first direction to show a first reflectionfeature;

FIG. 3B is a schematic illustration of an image of a passport bio-datapage illuminated from a second direction to show a second reflectionfeature;

FIG. 3C is a schematic illustration of an image of the passport bio-datapage of FIGS. 3A and 3B with no reflection features visible;

FIG. 4 is a chart illustrating the pixel intensity of a raw pixel dataset I_(PR) against distance d from the source of illumination;

FIG. 5 is a chart showing the final pixel intensity I_(PF) of the pixelsin the first set of pixel image data (as an example) against distance dfrom the source of illumination;

FIG. 6 is a chart showing pixel intensity I_(P) against apparentgreyness G (the response of the image capture device across the spectrumimaged) for decreasing pixel intensity;

FIG. 7 is a schematic example of the effect that gamma correction has ontext within an image;

FIG. 8 is a schematic illustration of a portion of the color sensorarray for an image capture device; and

FIG. 9 is a flow chart illustrating the preferred embodiment of thepresent invention.

DETAILED DESCRIPTION

Security documents such as passports, identification cards, and thelike, may often have either a matte or a shiny finish, and is unlikelyto be completely flat. During use, corners of plastic bio-data pages inpassports, for example, may bend, air bubbles and dirt may becometrapped within a laminate structure, or a surface material may be highlyreflective and shiny in appearance, all of which can create unwantedreflections, generally specular reflections, thus distorting thecaptured image. This may make machine readable text, such as OCR text,overt and covert security features difficult to read, and make automaticauthentication of the document and/or verification of the holderunreliable or impossible. For example, a bio-data page having a laminateconstruction with an extremely shiny surface may require additionalinspection by an operator if specular reflections distort the imagebeyond the capability of an automatic reader.

With the various constraints on security document imaging in mind, thereis a need for a method that allows the image taken by a standardsecurity document reader to be enhanced sufficiently that stray andunwanted reflections are no longer an issue, such that the document canbe authenticated reliably and accurately regardless of surface qualityor illumination conditions. Such a method may also find applications inother image capture techniques.

The present invention aims to address these issues by providing an imageenhancement method for an image capture device; the method comprisingthe steps of: illuminating an object placed on, in or adjacent to theimage capture device and capturing an image of the object from a firstposition to obtain a first set of raw pixel data; illuminating theobject placed on, in or adjacent to the image capture device, capturingan image of the object from a second position, different to the firstposition, to obtain a second set of raw pixel data, where each pixel inthe second set of raw pixel data corresponds to a pixel in the first setof raw pixel data representing a point on the object; calibrating eachof the first and second sets of raw pixel data using a first set ofimage calibration pixel data to create a first set of image pixel dataand using a second set of image calibration pixel data to create asecond set of pixel image data respectively, where each pixel in thesecond set of image calibration pixel data corresponds to a pixel in thefirst set of image calibration pixel data, and each pixel in the firstand second sets of image calibration pixel data corresponds to a pixelin each of the first and second sets of raw pixel data respectively; fora first pixel in the first set of image pixel data: i) calculating themodulus of the pixel intensity of said pixel in the first set of imagepixel data minus the pixel intensity of the corresponding pixel in thesecond set of image pixel data and comparing the modulus to apre-determined threshold value; ii) if the modulus is greater than thethreshold value, selecting the one of the pixel in the first set ofimage pixel data and the pixel in the second set of image pixel datawith the lowest pixel intensity as the output pixel; iii) if the modulusis less than the threshold value, determining the one of thecorresponding pixel in the first set of image calibration pixel data andthe corresponding pixel in the second set of image calibration pixeldata having the highest pixel intensity, and selecting the correspondingpixel in either the first set of image pixel data or the second set ofimage pixel data as the output pixel; and repeating steps i), ii) andiii) for at least a second pixel in the first set of image pixel dataand forming a set of final pixel data from the resulting output pixels.

The advantage of using such an approach is that only pixels representinga portion of an image in which a reflection is absent are used to makeup the set of final image data, thus ensuring that any image recoveredis of a high quality with reflections either attenuated or removed. Insome circumstances, reflections may in fact be separated, for example,specular reflections are removed but reflections from single colorfeatures remain. This is particularly advantageous for a securitydocument, such as an identity document or a fiduciary document, wherecovert or overt security features may be revealed as single colorreflections.

In one aspect of the present invention, the object is illuminated withvisible light, infra-red light or ultraviolet light. In another aspectof the present invention, when the object is illuminated with visiblelight, the object is illuminated with white light.

In another aspect of the present invention, the pixel intensity may havebalanced red-green-blue components. Alternatively, the pixel intensityhas un-balanced red-green-blue components. In this case, preferably thepixel intensity has a maximum red, green or blue component.

In another aspect of the present invention, the method may furthercomprise: for each pixel in the first and second sets of raw pixel data,measuring the intensity of single color reflections, and for pixelsrepresenting the same region of the object, selecting the pixel with thebrightest single color intensity; and including said pixel in a secondset of final image data.

In yet another aspect of the present invention, the method may alsofurther comprise the step of adjusting the first and second sets of rawimage data with a gamma correction.

In another aspect of the present invention, the image enhancement outputincludes the attenuation, separation or removal of reflections. In yetanother aspect of the present invention, the image enhancement outputincludes the attenuation, separation or removal of specular reflections.In another aspect of the present invention, the method may also furthercomprise: for each of the first and second sets of raw pixel data,compensating the intensity values of each pixel for ambient light. Inthis case, the method may also further comprise: creating a set ofambient pixel data by imaging the object under no illumination otherthan ambient light; and subtracting the set of ambient pixel data fromeach of the first and second sets of raw pixel data.

Preferably the object is a security document. More preferably the objectis an identity document or a fiduciary document. Yet more preferably theobject is one of a passport, an identification card or a driver'slicence. In each of these situations, preferably the image capturedevice is a security document reader.

In the present invention, the use of masking techniques are used tocreate an image of an object, such as a security document, that issubstantially free of unwanted reflections, as explained in furtherdetail below. The method comprises illuminating an object placed on, inor adjacent to an image capture device such as a security documentreader, so as to capture an image of the object from a first position toobtain a first set of raw pixel data. Next, a step of illuminating theobject placed on, in or adjacent to the image capture device, capturingan image of the object from a second position is carried out, where thesecond position is different to the first. This is to obtain a secondset of raw pixel data, where each pixel in the second set of raw pixeldata corresponds to a pixel in the first set of raw pixel datarepresenting a point on the object. Since there is a positionaldifference in the two images, calibrating each of the first and secondsets of raw pixel data is preferred. This is done using a first set ofimage calibration pixel data to create a first set of image pixel dataand using a second set of image calibration pixel data to create asecond set of pixel image data respectively, where each pixel in thesecond set of image calibration pixel data corresponds to a pixel in thefirst set of image calibration pixel data, and each pixel in the firstand second sets of image calibration pixel data corresponds to a pixelin each of the first and second sets of raw pixel data respectively. Atthis point, it is preferred to carry out an operation to mask thereflections detected to produce an enhanced image of the object.Starting with a first pixel in the first set of image pixel data, themodulus of the pixel intensity of the pixel in the first set of imagepixel data minus the pixel intensity of the corresponding pixel in thesecond set of image pixel data is calculated, and compared to themodulus to a pre-determined threshold value. If the modulus is greaterthan the threshold value, the one of the pixel in the first set of imagepixel data and the pixel in the second set of image pixel data with thelowest pixel intensity is selected as the output pixel. If the modulusis less than the threshold value, then the one of the correspondingpixel in the first set of image calibration pixel data and thecorresponding pixel in the second set of image calibration pixel datahaving the highest pixel intensity is determined. The correspondingpixel in either the first set of image pixel data or the second set ofimage pixel data is selected as the output pixel. These steps are thenrepeated for sufficient pixels in the first set of image pixel data, soas to render an image from a set of final pixel image data formed fromthe resulting output pixels.

By using such a masking technique described above, an imagesubstantially without reflections is revealed. Such a method isparticularly suitable for use with a security document. By attenuating,separating or removing unwanted reflections, in particular, specularreflections, the reliability of automated authentication of a securitydocument, either by text or overt security feature recognition or byrevelation of covert security features is improved.

In the following embodiments, the example of a security document andsecurity document reader is used. However, as described below, inalternative embodiments, the method of the present invention suitablefor use with other objects and image capture devices.

FIG. 1 is a schematic side view of a document reader in which anembodiment of the method of the present invention is carried out. Thedocument reader 1 is generally cuboid in shape, and comprises a housing2 in which first 3 and second 4 illumination sources and an imagecapture device 5 are positioned. The uppermost surface of the housing 2is formed from a glass platen 6, onto which a security document 7 may beplaced in order to be imaged. In this embodiment, in order to enableillumination of the document from a first and a second direction, thefirst 3 and second 4 lighting sources are positioned on either side ofthe image capture device 5, which is disposed centrally within thehousing 2 adjacent a wall 8 of the housing. Each illumination source 3,4 is provided with a linear array of light emitting diodes 9 a, 9 b, 9c, 9 d (only two of which are shown on each of the first 3 and second 4lighting sources for clarity), aligned to illuminate the entire surfaceof a security document 7 in contact with the glass platen 6. Lighttravels along the optical paths OP₁ and OP₂ to be incident on the glassplaten 6 and document 7, and reflected back to the surface of the imagecapture means 5. Non-limiting example optical paths are shown for thefirst illumination means 3 only. Second illumination source 4 mayinclude similar optical paths, although not illustrated. Preferably thelight emitting diodes emit light in the visible range of theelectromagnetic spectrum, with suitable LEDs being available from OsramOpto Semiconductors under the product code “TOPLED Ultra White 2PLCC”.The image capture means 5 is preferably a CMOS device, such as theMT9T001 ½ inch 3-megapixel digital image sensor, available from MicronTechnologies, Inc., located in Boise, Id., USA.

The document reader 1 illustrated in FIG. 1 is arranged so as to enablea method involving imaging a security document from a first and a seconddirection, where the second direction is different from the firstdirection. Using two different illumination directions allows images ofthe same point on the security document to be taken that yield differentoptical effects. This is generally illustrated in FIG. 2. FIG. 2 is aschematic side view of an optical defect in a security document givingrise to a specular reflection. Specular reflections may be anything thatincludes an optical glare reflecting back from a surface. Examples ofspecular reflection in a security document may be caused an unevenlaminate, uneven surface that is not optically flat, or the materialitself, such extra shiny laminates. In general, specular reflections aremirror or glass-like reflections. In the case of a security reader,there are artifacts or material properties in the security laminates ofa security document that cause bright white spots where the light fromthe light source(s) is reflected back to the image capture device. Asone example, optical defect 10 is present in the surface of a securitydocument 11, in this case, a bubble in a laminated bio-data pagestructure. Light from a first direction L₁ is incident on a first sideof the defect 10 and reflected R₁ onto an image capture device 12. Thisgives an image with a bright spot corresponding to reflection from thesurface of the defect 10 on which the light L₁ was incident. Light froma second direction L₂ is incident on a first side of the defect 10 andreflected R₂ onto an image capture device 12. This gives an image with abright spot corresponding to reflection from the surface of the defect10 on which the light L₂ was incident. These two images of the samesection of the security document 11 will appear to be subtly differentwhen compared to each other. When light reflected from the defect 10 isincident on the image capture device 12 different pixel intensities forthe same point on the security document are obtained as follows. Theimage capture device 12 contains an array of cells each of which has aone-to-one relationship with a pixel in an image of the securitydocument 11. When illuminated from a first direction with light L₁ thefirst set of raw pixel data obtained will contain a bright pixel at thepoint where the reflection R₁ is incident on the image capture device,at position A. When illuminated from a second direction with light L₂the second set of raw pixel data obtained will contain a bright pixel atthe point where the reflection R₂ is incident on the image capturedevice, at position B. When these two data sets are combined the darkestpixel (e.g. the pixel with the lowest pixel intensity measure for eachequivalent pixel) will be found in the second raw pixel data set atpoint A and in the first raw pixel data set at point B. A final imageformed from combining data based on these two data sets and using onlythe “darkest” pixels or pixels with the lowest pixel intensity measuredat each position on the security document such that reflections from thedefect 10 are effectively removed from this final image. This ispossible as each pixel in the second set of raw pixel data correspondsto a pixel in the first set of raw pixel data representing a point onthe security document.

This idea is illustrated further in FIGS. 3A, 3B and 3C. FIG. 3A is aschematic illustration of an image of a passport bio-data pageilluminated from a first direction L₁ to show a first reflectionfeature. A bio-data page is chosen in this example as typically this iscomprised of a multilayer laminated structure with at least one plasticor reflective layer or region on the page containing identityinformation about the passport bearer. However, the method describedbelow is equally suitable for any page or surface of a security documentthat requires imaging for bearer identification and/or documentauthentication to take place. The first reflection feature 13 is aspecular reflection obscuring a portion of text 14 on the bio-data page15. This is caused, for example, by a defect within the laminatedstructure of the bio-data page 15. FIG. 3B is a schematic illustrationof an image of a passport bio-data page illuminated from a seconddirection L₂ to show a second reflection feature. The second reflectionfeature 16 is a specular reflection obscuring a portion of thephotograph 17 of the holder of the bio-data page 15. This is caused, forexample, by the inclusion of a reflective covert security feature withinthe bio-data page 15. FIG. 3C is a schematic illustration of an image ofthe passport bio-data page of FIGS. 3A and 3B with no reflectionfeatures visible. This image is formed from a comparison of the twoimages in FIGS. 3A and 3B and using a masking technique to select pixelsrevealing a reflection free image.

In order to utilise a masking approach to its fullest extent, it ispreferably to ensure that the data collected in the first and second rawpixel data sets is as accurate as possible. To achieve this, two factorsmust be born in mind. Firstly, a document reader, such as a securitydocument reader, has a limited footprint due to size restrictions in theenvironment in which it is used, which would typically be a desk orcubicle at a border inspection point. This then places constraints onthe optical system within the document reader, as to enable illuminationof an entire security document placed on the reader lighting sourceoften need to be positioned adjacent a wall or corner of the housing ofthe document reader, as in the example given above. This causes avariation in the intensity of illumination of the security document withdistance away from the lighting means, and consequently a spatialdistribution of pixel intensity in image obtained. Secondly, the imagecapture device typically has an inherent non-linear response tointensity of illumination and color, leading to a variation between areal intensity for a particular shade and an ideal intensity for thesame shade. For a methodology that relies on being able to select thedarkest version of a pixel any discrepancy in illumination and/or colordefinition can have a detrimental effect on the data unless corrected.

FIG. 4 is a chart illustrating the pixel intensity of a raw pixel dataset I_(PR) against distance from the source of illumination d. Thisillustrates the effect of the spatial distribution of the light emittedfrom the lighting sources 3, 4, within the document reader 1 andincident on the security document 7. In this example, the lightingsource 3 is positioned adjacent d=0, such that the highest pixelintensity of raw pixel data I_(PR) occurs at this point. As the distanced away from the lighting source 3 increases the pixel intensity dropsoff substantially following a mean inverse square approximation. Therelationship shown is appropriate for two lighting sources, whereas fora greater number of lighting sources the resulting intensityrelationship is created using a mean inverse square approach resultingin a saddle-shaped intensity distribution. In this example,approximately half-way between the highest and lowest pixel intensitiesa reflection peak RP is seen. However, given the general noise withinthe data and the decreasing pixel intensity with distance d in thisposition it is likely that the reflection peak would be detected and thedarkest pixel method used successfully. However, a peak found at anincreased value of d, and therefore further into the region ofdecreasing pixel intensity may be harder to detect due to noise, andtherefore calibration of the raw pixel data to avoid this is advisable.

This variation in pixel intensity can be corrected using a set ofcalibration pixel data. Each of the first and second raw pixel data setswill have an intensity distribution similar to that shown in FIG. 4.Also shown on FIG. 4 is a line marked “WBG” representing whitebackground intensity. This is effectively the pixel intensity for aplain white background, such as a sheet of white paper or card, imagedusing the document reader 1. By allocating pixel intensity in the WBG torepresent the background value of pixel intensity in the raw pixel datasets, a set of calibration pixel data is created. When this set ofcalibration pixel data is combined with the raw pixel data in amathematical operation as shown in Equation 1 below the pixel image datais returned:

Output=(255×Input)/(WBG+c)  Equation 1

Output=output pixel intensity in pixel image dataInput=input pixel intensity in raw pixel data255=maximum intensity value allocated to the cell in the image capturedeviceWBG=intensity of corresponding pixel in calibration pixel datac=constant, greater than 0 and preferably 1, included to ensure that theOutput value is not infinite.

This operation is completed for both the first set of raw pixel data andthe second set of raw pixel data to obtain the first and second sets ofimage pixel data respectively. FIG. 5 is a plot showing the final pixelintensity I_(PF) of the pixels in the first set of pixel image data (asan example) against distance from the source of illumination d. It canbe seen that the background intensity is now substantially flat withincreasing distance, and the reflection peak RP seen clearly above thebackground intensity, allowing the darkest pixel to be chosen easily andaccurately. Since the two sets of raw pixel data are different,calibrating each of the first and second sets of raw pixel datacomprises using a first set of image calibration pixel data to create afirst set of image pixel data and using a second set of imagecalibration pixel data to create a second set of pixel image datarespectively. Each pixel in the second set of image calibration pixeldata corresponds to a pixel in the first set of image calibration pixeldata, and each pixel in the first and second sets of image calibrationpixel data corresponds to a pixel in each of the first and second setsof raw pixel data respectively.

The calibrated first and second sets of image pixel data may then beused to calculate a first set of final image data by using a maskingtechnique to select between corresponding pixels in an image of the samepoint on the image of the security document. The masking technique isintended to remove the effects of “background noise” to successfullyidentify pixels representing reflections within a particular set ofimage pixel data. To enable this, a thresholding operation is carriedout, where an arbitrarily chosen threshold is used to remove any noiseand to identify bright pixels, thus effectively creating a mask. Byconsidering the modulus of the difference in intensity between a firstpixel in the first set of image pixel data and the corresponding pixelin the second set of image pixel data, it will be obvious if areflection is present at such a position on the security document sincethe difference in intensity between the two pixels will be high. If thisis greater than the threshold (more than background noise) it can beassumed that the brighter pixel represents a reflection, hence toretrieve an image where no reflection is present the “darkest” or lowestintensity pixel of the two is used as the output pixel, and included inthe set of final image pixel data. However, if the modulus is below thethreshold, a reflection is absent, and hence another criterion must bechosen to determine which of the pixels in the first image pixel dataset or the second image pixel data set should be used in the set offinal image data.

In an embodiment of the present invention, this is done by assessing thevalue of the corresponding pixels in the first and second calibrationpixel sets. The calibration pixel data sets represent the backgroundintensity rather than any artefact of the security document beingimaged. In addition, there is greater noise in the pixels in aparticular raw pixel data set with increasing distance d from thelighting means. This has the effect that, when a pixel is chosen, merelychoosing the darkest pixel direct from the first image pixel data set orthe second image pixel data set may lead to a poor quality pixel beingchosen that yields little benefit in terms of final image quality.However, for regions where there is no reflection present, bydetermining which pixel in the first calibration data set and the secondcalibration set has the highest intensity and selecting the appropriatecorresponding pixel from either the first or second image pixel datasets, the brightest pixel giving the best quality final image possibleis yielded.

The operation may be done using a simple code loop as follows:

If |P₁ − P₂| > Threshold then if P₁ > P₂ P_(output) = P₂ else P_(outpu)t= P₁ else if W₁ > W₂ P_(output) = P₁ Else P_(output) = P₂Where P₁ is a pixel in the first set of image pixel data, P₂ is thecorresponding pixel in the second set of image pixel data. P_(output) isthe output pixel forming part of the final set of image pixel data. W₁is the pixel in the first set of calibration data corresponding to thepixel in the first set of image pixel data, and W₂ is the pixel in thesecond set of calibration data corresponding to the pixel in the secondset of image pixel data. W₁ and W₂ are therefore also correspondingpixels. The threshold is chosen arbitrarily, based on the maximumintensity of the image capture device used. For example, a CMOS devicewill typically have a maximum intensity of 255, hence a suitablethreshold to remove any background noise would be approximately 10% ofsuch a maximum, so around 30.

The code loop is repeated for as many times as is necessary to form aviable and useful image, which may be for all of the pixels in the firstimage pixel data set or for only a subset of these pixels. The resultingimage may be used in various authentication and verification processes,since any reflections in the original images are attenuated, reduced orseparated to the extent that further processing operations are reliableand reproducible.

However, as is evident from FIG. 6, it may be desirable to make afurther correction, such as a gamma correction, to take into account theinherent non-linear response to intensity of illumination and color,leading to a variation between a real intensity for a particular shadeand an ideal intensity for the same shade. FIG. 6 is a chart showingpixel intensity I_(P) against apparent greyness G (the response of theimage capture device across the spectrum imaged) for decreasing pixelintensity. In the centre of the response range the non-linear behaviourof the image capture device is at its most stark—with the greatestdeviation being either above (I₁) or below (I₂) the ideal intensityI_(IDEAL). The direction in which the deviation occurs is an artefact ofthe image capture device used, hence both upper and lower deviations areillustrated here for the purposes of explanation. In order to ensurethat the pixel intensity is as close to the ideal intensity as possiblea correction factor, often known as gamma correction, is used. Whenapplied to the pixel intensity at point A on curve I₂, the pixelintensity will be corrected to point A′ on the line I_(IDEAL), and whenapplied to the pixel intensity at point B on curve I₂, the pixelintensity will be corrected to point B′ on the line I_(IDEAL). Gammacorrection is an exponential function typically in the form shown inEquation 2 below:

V _(out) =AV _(in) ^(γ)  Equation 2

Where V_(out) is output, V_(in) is input, A is a constant and γ is thegamma exponential correction factor. A gamma correction is applied tothe first set of final image data if required to ensure that the dataquality in the first set of final image data is as high as possible,making it ideal as a starting point for further processing as part of adocument authentication process. FIG. 7 is a schematic example of theeffect that gamma correction has on text within an image. The upper lineof text contains a first group of letters 18 (all letter “A”)corresponding to low illumination intensity (i.e. at a large distance dfrom the lighting source) and thus appear all in a lighter shade ofgray, and a second group of letters 19 (all letter “A”) corresponding tohigh illumination intensity (i.e. at a small distance d from thelighting means) and thus appear all in a darker shade of gray. Bothgroups 18, 19 are without gamma correction. The lower line of textcontains a third group of letters 20 (all letter “A”) corresponding tolow illumination intensity (i.e. at a large distance d from the lightingsource) and a fourth group of letters 21 (all letter “A”) correspondingto high illumination intensity (i.e. at a small distance d from thelighting source), both with gamma correction. The effect of gammacorrection on an image is that, for the letters in the third group 20and fourth group 21, there is a greater contrast between individualshades, and a greater contrast between lighter shades (low illumination)and darker shades (bright illumination) in general (i.e. the contrastbetween the entire third group 20 and the entire fourth group 21).

Extraction of further image features, such as covert security featureshidden within the security document being imaged, or further correctionand enhancement of the raw image pixel data will now be described withrespect to further embodiments of the present invention.

Although in the above embodiment no distinction is made in relation tothe color of reflection under examination in a first further embodimentfeatures may be separated, attenuated, highlighted or removed byconsidering brightest single color intensities as a complement to themasking approach outlined above. For specular reflections RGB (red,green and blue intensity) values are typically balanced out creating abright white spot. However, for security features, often only one of theRGB values is maximised, since the feature is brighter in a single coloronly. So for the darkest only pixel approach outlined above, the pixelintensity has balanced red-green-blue components, since this correspondsto a white, specular reflection. For a security feature, the pixelintensity has un-balanced red-green-blue components. This may in fact bethat pixel intensity has a maximum red, green or blue component.

FIG. 8 is a schematic illustration of a portion of the color sensorarray for an image capture device. This is typical of a CMOS-type deviceused in the embodiment above. Sensors are grouped into groups of foureach comprising a red detector cell (R₁-R₈), a blue detector cell(B₁-B₈) and two green detector cells (G₁-G₈, G′₁-G′₈), representing acell having a one-to-one relationship with a pixel in a final image.Each sensor detects the appropriate color, with two green detectorsensors being included in each group to mimic the response of a humaneye. The color response of a reflection, i.e. determination of a pixelhaving the brightest single color intensity, is measured by consideringthe response of individual sensors within each group and adjacentsensors within each group and/or adjacent groups. For example, areflection with an intense blue component can be detected by merelylooking at the response of the blue detector sensors or the red andgreen detector sensors (for the presence or absence of a response) or bylooking at the response of adjacent blue detector sensors. For example,saturation of the blue B₂ sensor would result in the response of theblue B₁, B₃ and B₅ sensors being examined as strong response here wouldindicate a reflection peak. Consequently, by additionally measuring thecolor intensity of single color reflections by examining the colorresponse of the pixels in the first and second sets of raw pixel data,for pixels representing the same point on the security document, thepixel with the brightest single color intensity can be selected andincluded in the first set of final image data. As an alternative tousing the RGB color space it may be desirable to use a different colorspace, such as L*a*b*, since this mimics the natural response of the eyemore accurately than RGB space, which is advantageous when an operatorcompares images on a screen and the actual security document.

In the examples given above, no correction is required for the effectsof ambient lighting (i.e. light generated by the surrounds of thedocument reader rather than by the document reader), since typicallydocument readers are used in an enclosed situation, for example, byproviding a hood or lid covering the security document duringillumination. However, in some circumstances, such as when a documentreader is used in a booth or other open environment, it may be desirableto correct the image obtained by removing the intensity componentattributable to ambient light. In a further embodiment of the presentinvention, this is done by creating a set of ambient pixel data byimaging the security document under no illumination other than ambientlight. This may be achieved by placing the security document onto theglass platen 6 of the document reader 1 and without activating any ofthe lighting sources, capturing an image of the security document 7,thus creating the set of ambient pixel data. This set of ambient pixeldata is then subtracted from each of the first and second sets of rawpixel data. This may be done at the same time as other calibrationoperations, beforehand or afterwards, but before the first or secondsets of final image data are created.

As an example, the reflection removal technique was carried out using acommercially available security document reader, a QS1000 available inthe UK from 3M United kingdom PLC, 3M Centre, Cain Road, Bracknell,Berkshire, RG12 8HT, UK. Minor modifications were made to the reader tosplit the existing array of light-emitting diodes (LEDs) into twoseparate half-arrays to ensure that two separate lighting sources werecreated. This was done by physically re-wiring the circuit board andincluding additional code in the software controlling the illuminationto allow each half-array to be operated separately. In order to ensurethat there was a one-to-one identity between corresponding pixels in anydata sets obtained using either half-array, a mapping system was used touniquely identify pixels. Each pixel was allocated a unique identifierbased on its position with respect to an arbitrary x-axis correspondingto the front edge of the reader and an arbitrary-axis corresponding to aside edge of the reader, each identifier being of the format (x_(n),y_(n)).

To test the reflection removal technique, the following steps werecarried out, as shown in FIG. 9, a flow chart illustrating the preferredembodiment of the present invention. At step 101, a passport was openedto reveal the bio-data page, and placed face-down on the glass platen ofthe document reader. At step 102, the bio-data page was illuminatedusing the first half-array of LEDs from a first direction to capture thefirst raw pixel data set. At step 103, the bio-data page was illuminatedfrom a second direction, different to the first, using the secondhalf-array of LEDs to capture the second raw pixel data set. At step104, the first and second sets of raw pixel data were calibrated using aset of image calibration pixel data to create a first set of image pixeldata and a second set of pixel image data respectively. The set ofcalibration data was obtained initially when the document reader was setup to illuminate from two different directions by imaging a sheet ofwhite 80 gsm paper. At step 105, a first set of final image data wascalculated by comparing the first and second sets of image pixel dataand for pixels representing the same point on the object, selecting thepixel with the lowest pixel intensity; and including said pixel in thefirst set of final image data. This was done by using the followingloop:

For (x_(n), y_(n)) If |P₁ − P₂| > Threshold then if P₁ > P₂ P_(output) =P₂ else P_(outpu)t = P₁ else if W₁ > W₂ P_(output) = P₁ Else P_(output)= P₂Repeat for all x (x₁−x_(n)) and y, (y₁−y_(n)) to create the first set offinal image data comprising the OutputPixels for each (x, y). Once thefirst set of final image data was obtained, it was necessary to performa gamma correction exercise to ensure that any effects of the responseof the image capture device within the reader were minimised. To dothis, before initial use, the image capture device was calibrated usinga set of color reference targets available from X-Rite, 4300 44th St.SE, Grand Rapids Mich. 49512, USA. The color reference targets comprisea set of greyscale targets with known RGB values, which in conjunctionwith image calibration software allow a matrix of γ values to becalculated at certain points in the response of the image capturedevice. This matrix of γ values was then applied to the first set offinal image data to correct for any inherent response behaviour in theimage capture device.

Although the technique was carried out using a passport bio-data page,it is also possible to image any other page of a passport, anidentification card or a driver's licence, as examples of identitydocuments. Other security documents, such as fiduciary documents (forexample, credit or bank cards) may also be imaged using this technique.In the above example, the processing to create the various sets of datais carried out within the FPGA (field-programmable gate array) of thedocument reader. However, this is merely a matter of preference, and theprocessing could alternatively be carried out in an ASIC(application-specific integrated circuit) if desired.

In the above embodiments, images are captured from different positions,such as from different angles. This is dictated by the physicalconstruction of a passport reader, which has a dedicated footprintlimited in size due to the constraints of the areas in which suchreaders are often situated. A typical full page passport reader has anapproximate base size of 160 mm×200 mm and a height of approximately 190mm. The lighting sources are typically placed adjacent a side wall,approximately 50 to 70 mm away from the wall, resulting in a typicalangle of illumination in the range of 10° to 60° and typically around40° to 50° (where the angle is measured at the surface of the securitydocument being illuminated). This is relatively wide angle illuminationcompared with other image capture devices, such as cameras. Consequentlythe first and second positions from which the security document isilluminated and the images captured from are determined by the first andsecond illumination angle created by the position of the first andsecond lighting sources. However, it is possible to create illuminationand/or image capture from different relative positions without using twoseparate lighting sources. For example, a single image capture devicecan be replaced with two or more image capture devices, in conjunctionwith a single lighting source. Alternatively, further optical paths canbe created from either a single or multiple light source(s) usinglenses, mirrors or prisms, with each optical path yielding a relativeposition from which the security document may be illuminated or an imagecaptured. Creating different relative positions from which to illuminatethe security document or from which to capture images of the securitydocument may also be achieved by moving the security document and/or theimage capture device relative to each other. This could be using a motoror vibrating either the security document (for example, by moving theglass platen) or the image capture device at a fixed frequency. Creatingmultiple relative positions from which either the security document canbe illuminated or from which images can be captured is particularlyuseful for identifying holographic features. Further options could alsoinclude the use of plenoptic light field cameras or the use of microlensarrays to create multiple images that appear to be imaged from multipleangles.

In the embodiments described above, the approach of the presentinvention is applied to a security document reader to address issuesinvolving reflections in security documents. However, the techniques maybe used with other image capture devices (including, but not limited to,cameras—whether digital, video or otherwise—CMOS and CCD devices, mobilephones and other hand held devices, optical scanners, including flat bedscanners and other equipment that is capable of capturing an image) inwhich reflections arising from optical or physical defects orinconsistencies in the object being imaged occur. In the embodimentsdescribed above, the security document may be replaced by an object, forexample a different type of document (in the case of a scanner), or aperson or landscape scene (in the case of a camera). This may or may notbe in contact with the image capture device, and the angle ofillumination may be relatively narrow compared with the example of apassport reader above. However, illumination of the object or capture ofan image of the object from at least two positions enables the darkestonly pixel technique to be applied to remove reflections in images ofthe object. The code loops described above also apply equally well toother object types and image capture devices, since images of an objectfrom different positions will always yield at least one image in which areflection is present at a certain point and at least a second imagewhere a reflection is absent at the same point, hence there will alwaysbe one bright and one dark corresponding pixel.

The present invention has now been described with reference to severalembodiments thereof. The foregoing detailed description and exampleshave been given for clarity of understanding only. No unnecessarylimitations are to be understood therefrom. All patents and patentapplications cited herein are hereby incorporated by reference. It willbe apparent to those skilled in the art that many changes can be made inthe embodiments described without departing from the scope of theinvention. Thus, the scope of the present invention should not belimited to the exact details and structures described herein, but ratherby the structures described by the language of the claims, and theequivalents of those structures.

What is claimed is:
 1. An image enhancement method for an image capturedevice comprising: illuminating an object placed on, in or adjacent tothe image capture device and capturing an image of the object from afirst position to obtain a first set of raw pixel data; illuminating theobject placed on, in or adjacent to the image capture device, capturingan image of the object from a second position, to obtain a second set ofraw pixel data, where each pixel in the second set of raw pixel datacorresponds to a pixel in the first set of raw pixel data representing apoint on the object; calibrating each of the first and second sets ofraw pixel data using a first set of image calibration pixel data tocreate a first set of image pixel data and using a second set of imagecalibration pixel data to create a second set of pixel image data,wherein each pixel in the second set of image calibration pixel datacorresponds to a pixel in the first set of image calibration pixel data,and each pixel in the first and second sets of image calibration pixeldata corresponds to a pixel in each of the first and second sets of rawpixel data respectively; for a first pixel in the first set of imagepixel data: i) calculating the modulus of the pixel intensity of saidpixel in the first set of image pixel data minus the pixel intensity ofthe corresponding pixel in the second set of image pixel data andcomparing the modulus to a pre-determined threshold value; ii) if themodulus is greater than the threshold value, selecting the one of thepixel in the first set of image pixel data and the pixel in the secondset of image pixel data with the lowest pixel intensity as the outputpixel; iii) if the modulus is less than the threshold value, determiningthe one of the corresponding pixel in the first set of image calibrationpixel data and the corresponding pixel in the second set of imagecalibration pixel data having the highest pixel intensity, and selectingthe corresponding pixel in either the first set of image pixel data orthe second set of image pixel data as the output pixel; and repeatingsteps i), ii) and iii) for at least a second pixel in the first set ofimage pixel data and forming a set of final pixel data from theresulting output pixels.
 2. The image enhancement method of claim 1,wherein the object is illuminated with visible light, infra-red light orultraviolet light.
 3. The image enhancement method of claim 1, whereinwhen the object is illuminated with visible light, the object isilluminated with white light.
 4. The image enhancement method of claim3, wherein the pixel intensity includes balanced red-green-bluecomponents.
 5. The image enhancement method of claim 3, wherein thepixel intensity includes un-balanced red-green-blue components.
 6. Theimage enhancement method of claim 3, wherein the pixel intensityincludes a maximum red, green or blue component.
 7. The imageenhancement method of claim 1, further comprising: for each pixel in thefirst and second sets of raw pixel data, measuring the intensity ofsingle color reflections, and for pixels representing the same region ofthe object, selecting the pixel with the brightest single colorintensity; and including said pixel in a second set of final image data.8. The image enhancement method of claim 1, further comprising:adjusting the first and second sets of image pixel data with a gammacorrection.
 9. The image enhancement method of claim 1, wherein theimage enhancement comprises the attenuation, separation or removal ofreflections.
 10. The image enhancement method of claim 1, wherein theimage enhancement comprises the attenuation, separation or removal ofspecular reflections.
 11. The image enhancement method of claim 1,further comprising: for each of the first and second sets of raw pixeldata, compensating the intensity values of each pixel for ambient light.12. The image enhancement method of claim 10, further comprising:creating a set of ambient pixel data by imaging the object under noillumination other than ambient light; and subtracting the set ofambient pixel data from each of the first and second sets of raw pixeldata.
 13. The image enhancement method of claim 1, wherein the object isa security document.
 14. The image enhancement method of claim 13,wherein the object an identity document or a fiduciary document.
 15. Theimage enhancement method of claim 13, wherein the object is a passport,an identification card, or a driver's license.
 16. The image enhancementmethod of claim 13, wherein the image capture device is a securitydocument reader.
 17. The image enhancement method of claim 1, whereinthe first position is different from the second position.
 18. The imageenhancement method of claim 17, wherein the first position is at a firstangle relative to the object, and wherein the second position is at asecond angle relative to the object.